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IN THE UNITED STATES PATENT AND TRADEMA RK OFFICE 

Inventor (s) : Winfried KOENIG et al . 

To Be Assigned 
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Serial No. 
Filed 
For 



METHOD AND DEVICE FOR MONITORING THE 
INTERIOR AND SURROUNDING AREA OF A VEHICLE 



Examiner : To Be Assigned 

Art Unit : To Be Assigned 



Assistant Commissioner 

for Patents 
Washington, D.C. 20231 

PRELIMINARY AMENDMENT 

SIR: 



Kindly amend the above- identified application before 
examination, as set forth below. 



IN THE TITLE : 

Please replace the title with the following new 

title : 

--METHOD AND DEVICE FOR MONITORING THE INTERIOR AND 
SURROUNDING AREA OF A VEHICLE--. 



IN THE DRAWINGS ; 

Please amend the drawings as indicated on the 
attached red-marked sheets. 



IN THE SPECIFICATION : 

Please amend the specification as follows. 

On page 1, delete lines 1-3, and insert: 
- ^ FIELD OF THE INVENTION 

The present invention relates to a method and device for 
monitoring the interior and surrounding area of a vehicle. 



BACKGROUND INFORMATION 
The article '"Die 



On page 1, line 5, change ^^in" to --published in the 
October 1998 issue of--. 

On page 1, line 6, delete ^^October 1998 issue, page 

8 ; - - . 

On page 1, line 9, delete ^^then" . 

On page 1, line 10, change image, which are then^' 
to --Then, the road signals are--. 

On page 1, line 15, change ^^which" to --that--. 

On page 1, line 16, delete 'Vhich" . 

On page 1, line 18, change "known from" to -- 
described in PCT Patent No.--. 

On page 1, delete line 21, and insert: 
-- SUMMARY -- . 

On page 1, line 22, change "By contrast, the" to 

--The-- . 

On page 1, delete line 23, and insert: 
- -has the- - . 

On page 1, line 24, change "as well as" to 

- -and- - . 

On page 2, line* 4, delete "particularly". 

On page 2, line 8, change "only" to --is--. 

On page 2, line 9, change "which" to --that--. 

On page 2, line 11, delete "particularly". 
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On page 2, line 12, change "pref erably^' to --for 

example-- . 

On page 2, line 18, change "of the interior; a" to 
--the interior. A--. 

On page 2, line 24, delete "in particular". 

On page 2, line 25, change "In particular" to --For 

example- - . 

On page 2, line 29, change "Thus" to --Thus,--. 

On page 2, line 32, change "source" to --source,--. 

On page 2, line 33, change "which" to --that--. 

On page 3, line 4, change terms, which" to --terms. 

This-- . 

On page 3, line 6, delete ^^particularly^' , and change 
''carry out" to --perform--. 

On page 3, line 7, change "switching" to --switch--. 
On page 3, line 10, change "in particular" to --for 

example- - . 

On page 3, line 18, change "In particular" to --For 

example- - . 

On page 3, line 19, change "carried out" to 
- -performed- - . 

On page 3, delete line 30, and insert: 
--for example his eyes, the road markings, and--. 



On page 3, line 31, delete ^^respectively , . 

On page 3, line 32, change ^^markings; this is 
because this" to --markings. This--. 

On page 3, line 35, change "which" to --that--. 

On page 3, line 36, change ^^Thus, because" to 
--Since--, and delete "additional". 

On page 3, line 37, change "safety compared to the 
related art" to --safety, as compared to conventional methods 
and devices- - . 

On page 4, line 1, change "achieved" to 
- -increased- - . 

On page 4, line 3, change "some" to --a--. 

On page 4, line 12, change "Because" to --Since--. 

On page 4, line 13, delete "because". 

On page 4, line 28, after "for" insert --an--. 

On page 4, line 30, change "namely" to --for 

example- - . 

On page 4, line 32, change "heater" to --heater,--, 
and change "using" to --sitting--. 

On page 4, line 33, change "In particular" to --For 

example- - . 

On page 5, line 5, change ^^pref erably" to --for 

example-- . 
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On page 5, line 7, change ^^unclear" to --unclear,--. 

On page 5, line 15, change ^^carrying" to 
- -performing- - . 

On page 5, line 16, delete ^^out'^ . 

On page 5, line 18, delete ^^so as". 

On page 5, line 19, delete ^^of" (second occurrence). 

On page 5, line 20, change ^^In particular" to --For 

example- - . 

On page 5, line 23, change ^^pref erably" to --for 

example- - . 

On page 6, line 1, change ^^as" to --as, for 
example , - - . 

On page 6, line 4, delete ^^particularly". 

On page 6, line 8, change ^^objects" to --objects,--. 

On page 6, line 15, delete "of". 

On page 6, line 20, delete "particularly". 

On page 6, delete lines 32-37, and insert: 
-- BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 shows an arrangement of a device according to the 
present invention in a motor vehicle. 

Figure 2 shows a flow chart of a first embodiment of a method 
according to the present invention. 

Figure 2a shows a first process step of a method according to 
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the present invention. 



Figure 2b shows a second process step of a method according to 
the present invention. 

Figure 2c shows an evaluation method according to the present 
invention . 

Figure 3 shows a flow chart for a second embodiment of a 
method according to the present invention. 

Figure 4 shows a first embodiment of a device according to the 
present invention. 

Figure 5 shows a second embodiment of a device according to 
the present invention. 

Figure 6 shows a third embodiment of a device according to the 
present invention. 

Figure 7 shows a fourth embodiment of a device according to 
the present invention. 

Figure 8a shows a first embodiment of a deviation mirror 
according to the present invention. 

Figure 8b shows a second embodiment of a deviation mirror 
according to the present invention. 

DETAILED DESCRIPTION -- . 

On page 7, delete lines 1-14. 

On page 7, line 18, change ^^13, a" to --13. A--, 
and change ^^leading" to --leads--. 



On page 7, line 21, change "opposite side" to - 
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side--, and after '"10" insert --opposite--. 

On page 7, line 23, change ''Second" to --Therefore, 

second- - . 

On page 7 , line 24, delete ''therefore only" . 

On page 7, line 31, change "preferably embodied as" 
to --embodied as, for example,--. 

On page 7, line 33, change "In particular" to --For 

example- - . 

On page 7, line 37, change "unit" to --unit,--. 

On page 8, line 2, change "either be arranged in" to 
--be arranged in, for example, either--. 

On page 8, line 3, delete "or in". 

On page 8, line 4, delete "in". 

On page 8, line 5, change "a preferred" to --an--. 

On page 8, line 17, change "Preferably" to 
--Therefore,--, and delete "therefore". 

On page 8, line 18, change "arranged" to --arranged, 
for example , - - . 

On page 8, line 34, change "unit, a" to --unit. 
A--, and change "being output" to --is output--. 

On page 9, line 2, change "35, in which" to --35. 
In second process step 34,--. 

On page 9, line 5, change "previously," to 
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--previously. Image information regcirding the vehicle 
interior is determined--. 

On page 9, line 8, delete ''obtained", and change 
"also output" to --obtained and output--. 

On page 9, line 9, change ''media, the second output 
being in particular" to --media. The second output is--. 

On page 9, line 12, delete ''in". 

On page 9, line 13, delete "particular". 

On page 9, line 31, delete "of". 

On page 9, line 33, change "carried out" to 
- -performed- - . 

On page 10, line 8, change "which" to --that--. 

On page 10, line 9, change "device and of" to 
--device, . 

On page 10, line 10, change "33" to --33,--. 

On page 10, line 34, change "carried out" to 
- -performed- - . 

On page 11, line 13, change "preferably" to --for 

example- - . 

On page 11, line 23, change "carried out" to 
- -performed- - . 

On page 11, line 25, after "60," insert --and--. 

On page 11, line 26 change "being" to --is--. 
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On page 12, line 16, change ''carried ouf to 
- -performed- - . 

On page 12, line 27, after ''vehicle'' insert --and-- 

On page 13, line 5, change "carried out" to 
--performed-- . 

On page 13, line 9, change "preferred" to --an--. 

On page 13, line 14, change "Particularly" to --For 
example, . 

On page 13, line 22, change "having" to --that 

has- - . 

On page 13, line 24, change "as" to --as, for 
example, - - . 

On page 14, line 11, change "housing which is 
preferably" to --housing, which, for example, is--. 

On page 14, line 25, change "a preferred" to --an-- 

On page 14, line 28, change "in" to --of--. 

On page 14, line 34, delete "preferably". 

On page 14, line 36, change "carried out" to 
- -performed- - . 

On page 15, line 2, change "captured, this" to 

y 

--captured. This-- . 

On page 15, line 3, change "being" to --is--. 

On page 15, line 7, change "Preferably the" to 
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--The--, and change "as'^ to --as, for example,--. 

On page 15, line 14, change "through infra-red 
radiation" to --infra-red radiation through--. 

On page 15, line 16, change "Thus in particular'' to 
--Thus, for example,--. 

On page 15, line 18, delete ^^thus" . 

On page 15, line 28, change "interior" to 
--interior, -- . 

On page 15, line 30, change ^^100" to --100,--. 

On page 16, line 4, change "mean" to --include--. 

On page 16, line 6, change "which mean that" to 
--since--, and after "seen" insert --by--. 

On page 16, line 7, change "mean" to --include--. 

On page 16, line 8, change "as" to --as, for 
example , - - . 

On page 17, line 1, change ^^a preferred" to --an--. 

On page 17, line 21, change "as" to --as, for 
example, - - . 

On page 17, line 30, change "carrying out" to 
- -performing- - . 

On page 18, line 13, change "provided," to 
- -provided that - - . 

On page 18, line 32, change "is" to --are--. 
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On page 19, line 2, change '"cameras" to 
- -cameras , - - . 

On page 19, line 3, change ''another" to 
- -another, - - . 

On page 19, line 21, change ''thanks" to --due--. 

IN THE ABSTRACT ; 

Please amend the Abstract as follows. 

Delete line 1, and insert: 
ABSTRACT -- . 

Line 3, change "the area surrounding" to --a 
surrounding area- - . 

Line 4, change "is" to --are--. 

Line 5, change "carrying out" to 
--performing--, and change "(10) having" to --that has--. 

Line 6, delete "(103)", and change "the direction of 
the" to --a direction of the surrounding--. 

Line 7, change "surrounding" to --of--, and change 
"the road, and having" to --a road, and has--. 

Line 8, delete "(108)", and change "the direction" 
to --a direction--. 

Line 9, delete "(110)". 

IN THE CLAIMS : 

On page 20, delete line 1, and insert: 
- - What Is Claimed Is : - - . 
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Please cancel claims 1-26, without prejudice. 



Please add the following new claims: 
26. (New) A method for monitoring an interior of a motor 
vehicle and a surrounding area of the motor vehicle, 
comprising : 

(1) capturing an image of at least part of the 
surrounding area of the motor vehicle by a camera device; 

(2) capturing an image of at least part of the 
interior of the motor vehicle by the camera device, the 
steps (1) and (2) being performed alternately; and 

(3) transmitting the images obtained in steps (1) 
and (2) to a processing unit. 

27. (New) The method according to claim 26, wherein: 

the at least part of the surrounding area of the 
vehicle is in a direction of travel. 

28. (New) The method according to claim 26, wherein: 

the at least part of the interior of the vehicle 
includes parts of a body of a driver. 

29. (New) The method according to claim 26, wherein step (1) 
includes illuminating the interior of the vehicle by a 
radiation source, the radiation source emitting a radiation at 
least substantially invisible to the human eye. 

30. (New) The method according to claim 29, wherein: 

the radiation source is an infra-red radiation 
source . 

31. (New) The method according to claim 26, wherein step (2) 
includes : 

superimposing the at least part of the interior 
of the vehicle visible to the camera device on the 
at least part of the surrounding area of the vehicle 
visible to the camera device; and 
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determining the image of the at least part of 
the interior of the vehicle by subtracting the image 
of the at least part of the surrounding area. 

32. (New) The method according to claim 26, wherein: 

only an image of an area surrounding the motor 
vehicle visible to the camera device is captured in the 
step (1) ; and 

only an image of the interior of the motor vehicle 
visible to the camera device is captured in the step (2) . 

33. (New) The method according to claim 32, wherein: 

switching back and forth between the step (1) the 
step (2) is accomplished via at least one light valve. 

34. (New) The method according to claim 33, wherein: 

the at least one light valve is an electro-optical 
light valve. 



35. (New) The method according to claim 26, wherein an image 
captured is only a partial area of a maximum image that may be 
captured by the camera device, the partial area of the maximum 
image including at least one of image rows, image columns, and 
image pixels, and wherein the method further comprises: 

switching back and forth between capturing a partial 
area of the interior and a partial area of the 
surrounding area; 

processing by the processing unit the partial areas 
captured; and 

capturing a next partial area. 

36. (New) The method according to claim 26, further 
comprising : 

capturing a face of a driver, the face including 
eyes of the driver. 

37. (New) The method according to claim 26, further 
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comprising : 

capturing at least one of road markings and a 
position of the vehicle relative to the road markings. 

38. (New) The method according to claim 36, further 
comprising : 

evaluating at least one of the face of the driver 
and a position of the vehicle relative to road markings 
to determine at least one of whether the eyes of the 
driver are open and whether the vehicle is moving beyond 
a predefined area of the road markings; and 

issuing at least one of a visual warning and an 
audible warning based on the evaluation. 

39. (New) The method according to claim 26, further 
comprising : 

capturing road signs. 

40. (New) The method according to claim 26, further 
comprising : 

determining at least one of a number of people in 
the vehicle and a seat occupancy. 

41. (New) The method according to claim 40, further 
comprising : 

deactivating at least one of an airbag and a seat 
heater of a corresponding seat when the corresponding 
seat is one of empty and occupied by a child seat. 

42. (New) The method according to claim 26, further 
comprising : 

capturing lip movements of a person in the vehicle 
to support a speech input system. 

43. (New) The method according to claim 42, wherein: 

the person is a driver of the vehicle. 
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44. (New) A device for monitoring an interior of a motor 
vehicle and a surrounding area of the motor vehicle, 
comprising : 

a camera device configured to alternately capture an 
image of at least part of the interior of the motor 
vehicle and an image of at least part of the surrounding 
area of the motor vehicle; and 

a processing unit connected to the camera device, 
the images captured by the camera device transmitted to 
the processing unit. 

45. (New) The device according to claim 44, wherein: 

a first beam path of the camera device points in a 
direction of a road in front of the vehicle; and 

a second beam path of the camera device points in a 
direction of the interior. 

46. (New) The device according to claim 45, wherein: 

the second beam path of the camera device points in 
a direction of a driver in the interior. 

47. (New) The device according to claim 44, further 
comprising : 

an illumination unit configured to emit a radiation 
at least substantially invisible to the human eye, the 
illumination unit controlled by the processing unit. 

48. (New) The device according to claim 47, wherein: 

the radiation is infra-red radiation. 

49. (New) The device according to claim 44, further 
comprising : 

an infra-red filter arranged in the camera device. 

50. (New) The device according to claim 49, wherein: 

the infra-red filter is arranged in the second beam 
path in the direction of the interior. 
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51. (New) The device according to claim 44, further 
comprising : 

at least one light valve arranged in the camera 
device . 

52. (New) The device according to claim 51, wherein: 

the at least one light valve is a liquid crystal 

cell . 

53. (New) The device according to claim 44, further 
comprising : 

at least one deviation mirror arranged in the camera 
device . 

54. (New) The device according to claim 53, wherein: 

the at least one deviation mirror is semi- 
transparent . 

55. (New) The device according to claim 54, wherein: 

the at least one deviation mirror is one of concave 
and convex. 

56. (New) The device according to claim 44, wherein: 

the camera device has a single camera. 

57. (New) The device according to claim 56, wherein: 

the single camera is one of a CCD camera and a CMOS 
camera . 

58. (New) The device according to claim 44, wherein: 

the camera device has at least two cameras for 
capturing images stereoscopically . 

59. (New) The device according to claim 44, further 
comprising : 

at least one of visual output units and acoustic 
output units connected to the processing unit, the at 
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least one of visual output units and acoustic output 
units configured to issue a warning to a driver when one 
of eyes of the driver are closed and the vehicle is about 
to move beyond a marked area of a road. 

60. (New) The device according to claim 44, wherein: 

the camera device is one of arranged in an upper 
part of a windshield and integrated into a roof of the 
vehicle . 

61. (New) The device according to claim 53, further 
comprising : 

an adjustment device configured to adjust the at 
least one deviation mirror so that at least eyes and lips 
of a driver can be seen in the image of the interior of 
the vehicle captured by the camera device. 

Remarks 

This Preliminary Amendment cancels, without 
prejudice, claims 1-25 in the underlying PCT Application No. 
PCT/DEOO/01426 and adds new claims 26-61. The new claims 
conform the claims to U.S. Patent and Trademark Office rules 
and do not add new matter to the application. 

The above amendments to the title, drawings, 
specification and abstract conform the title, drawings, 
specification and abstract to U.S. Patent and Trademark Office 
rules, and do not introduce new matter into the application. 

The underlying PCT Application No. PCT/DEOO/01426 
includes an International Search Report, dated September 21, 
2000. The Search Report includes a list of documents that 
were uncovered in the underlying PCT Application. A copy of 
the Search Report is included herewith. 

It is respectfully submitted that the subject matter 
of the present application is new, non-obvious, and useful. 
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Prompt consideration and allowance of the application are 
respectfully requested. 



Respectfully submitted, 



Dated: ? /g^ /o [ By: ^ ^.^^^---^ 

Richard L. May^:^ 
F^eg. No. 22,4 90 

PCENYON & KENYON 
One Broadway 
New York, NY 10 0 04 
(212) 425-7200 
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METHOD AND DEVICE FOR MONITORING THE INTERIOR AND SURROUNDING 

AREA OF A VEHICLE 

Background of the Invention 

The invention is based on a method according to the definition 
of the species set forth in the main claim. The article "Die 
neuen Augen des Autos, Limousinen lernen lesen [Cars Get New 
Eyes, Limos Learn to Read]" in the journal Bosch Zunder, 
October 1998 issue, page 8, describes a method in which the 
area in front of the driver surrounding the vehicle is 
monitored by two video cameras. The image captured by the 
cameras is then evaluated with regard to road signs that can 
be detected in the image, which are then displayed to the 
driver via a display unit. In addition, the system captures 
the path of the road in order to control the direction of the 
headlamps so that the light cone falls on the road. If the car 
enters the shoulder, an audible and/or visual warning is 
triggered. Furthermore, a method which measures brain 
activity, in particular of the driver of a vehicle, and which 
triggers an alarm if there are deviations from the normal 
awake status, is known from WO 93/21615. Herein, measurements 
are taken via electrodes placed on the driver's head. 

Advantages of the Invention 

By contrast, the method according to the present invention 
having the features set forth in the main claim has the 
advantage that the interior of as well as the area surrounding 
a vehicle can be captured using just one camera device. In 
particular, this is feasible because the interior of the 
vehicle and the area surrounding the vehicle are captured 
alternately. Provided the system alternates sufficiently 
quickly between capturing the interior and capturing the 
surrounding area, loss of information arising from switching 
back and forth may be ignored, and just one camera device as 
opposed to two is required for the interior and the area 
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surrounding the vehicle. Furthermore, only one processing unit 
for processing the image information obtained is required. 

Moreover, it is particularly advantageous that the interior of 
the vehicle is illuminated by a radiation source that is at 
least largely invisible to the human eye. This has the 
advantage that during night driving, when as a general rule 
the interior of a vehicle is not lit or only poorly lit, the 
interior can nevertheless be monitored by a camera which is 
sensitive to radiation emitted by the radiation source. Herein 
it is particularly advantageous to use an infra-red radiation 
source, preferably one or more infra-red light-emitting 
diodes. This does not distract the driver, unlike a visible 
source . 

Furthermore, it is advantageous to obtain an image of the 
interior from a superimposition of an image of the surrounding 
area and of the interior; a processing unit subtracts an image 
of the exterior only from this superimposition. As a result, 
when the system alternates between capturing the surrounding 
area and the interior, there is no need to interrupt recording 
of the exterior, because the system simply interrupts 
recording of the interior. As a result, there is no need for 
optical interrupt elements, such as in particular mechanical 
shutters or mirrors. In particular, if the interior is 
illuminated by an infra-red radiation source and the image of 
the interior is captured via an infra-red filter, an image of 
the interior is essentially only captured if the infra-red 
radiation source is activated. Thus alternating monitoring of 
the interior and the surrounding area is feasible via a 
switch-off /switch-on sequence for the infra-red radiation 
source provided the camera device has a further beam path 
which extends into the area surrounding the vehicle and can 
capture the area surrounding the vehicle. 

Furthermore, it is advantageous to capture only the visible 
part of the surrounding area in a first process step, and to 

2 
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capture only the visible part of the interior in a second 
process step. Thus it is not necessary for the images of the 
interior and the surrounding area to be separated in 
processing terms, which means the processing unit in which the 
image data is evaluated does not have to be especially 
powerful. Herein, it is particularly advantageous to carry out 
the switching over between capturing the part of the 
surrounding area visible to the camera and capturing the part 
of the interior visible to the camera via an electro-optical 
light valve, in particular via a liquid crystal cell, which 
can be switched back and forth between a transparent mode and 
an absorptive mode based on a signal applied. 

Furthermore, it is advantageous when switching back and forth 
between capturing image signals from the surrounding area and 
image signals from the interior to switch back and forth as 
soon as partial areas of the maximum area that can be captured 
by the camera device have been captured. In particular, 
switching back and forth may be carried out after image 
columns or image rows have been captured or after groups of 
pixels have been captured. As the image data also has to be 
transmitted to the processing unit and processed there, this 
method has the advantage that it allows quicker switching back 
and forth between capturing the interior and the exterior, so 
that the shift between two captured images, e.g., of the 
exterior, which is based on the movement of the vehicle, is 
reduced . 

Furthermore, it is advantageous to capture the driver's face, 
in particular his eyes, as well as the road markings and, 
respectively, the position of the vehicle relative to the 
road markings; this is because this information can be used to 
determine whether the driver may have fallen asleep and may 
therefore be driving in an uncontrolled manner, and can be 
used to activate a warning device which wakes up the driver. 
Thus, because the driver's face is also captured, additional 
safety compared to the related art, in which a camera device 

3 
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only captures the road markings, is achieved. For example, 
when the vehicle is traveling in a straight line for a long 
period, the vehicle may travel for some considerable time 
within the road markings, even though the driver has already 
been asleep for a number of seconds. Using the method 
according to the present invention, it is possible to detect 
that the driver has fallen asleep in a case of this kind. 

If the camera is used for monitoring, it is not necessary to 
place electrodes on the driver's body, which is necessary in 
the case of the method in which the driver's brain waves are 
monitored. Because electrodes of this kind may be cumbersome 
and may limit the driver's freedom of movement, and because 
the driver may also forget to put them on when he starts 
driving or may deliberately not put them on because they are 
uncomfortable, a warning indicating that the driver has fallen 
asleep is easier to implement and less unpleasant for the 
driver to use. Furthermore, the method according to the 
present invention has the advantage that as well as monitoring 
the interior, the system can capture road signs in the area 
surrounding the vehicle and can therefore alert the driver, 
for example, to warning signs or speed limit signs via a 
visual or acoustic output unit. 

Furthermore, it is advantageous to determine the number of 
people in the vehicle or, respectively, the seat occupancy. 
This information can be used, for example, to control the 
chassis so as to compensate for uneven load in the vehicle if, 
for example, people are only sitting on the left side of the 
vehicle, namely the driver and one person behind him. 
Furthermore, this information can be use to control a seat 
heater which is only activated if someone is actually using on 
the seat. In particular, it is possible to determine whether a 
seat is occupied or is occupied by a child seat, as it is 
advantageous that deployment of an airbag can be blocked if a 
seat is unoccupied or is occupied by a child seat. As a 
result, unnecessary deployments of an airbag can be avoided if 
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the seat is unoccupied, and injury to a child by an airbag can 
be prevented if the seat is occupied by a child seat. 

Furthermore, it is advantageous to also capture the lip 
movements of a predefinable person in the vehicle, preferably 
the driver, in order to support a speech input system. If, for 
example, during speech input it is unclear which command has 
been input due to driving noise, the driver's lip movements 
can be captured by the camera device and evaluated so as to 
check the speech input. This is possible, for example, if the 
lip movements are analyzed to determine whether the syllables 
that correspond to the lip movements captured are contained in 
the command understood by the speech input unit. If the speech 
input unit cannot make unambiguous assignments based on what 
it has understood, this can possibly be achieved by carrying 
out a comparison with the lip movements. 

Furthermore, it is advantageous to provide a device so as to 
allow capturing of the area surrounding the vehicle and of the 
vehicle interior. In particular, it is advantageous to design 
a camera device so that one beam path points in the direction 
of the interior and one beam path points in the direction of 
the road, preferably in the direction of travel, because as a 
general rule from the driver's point of view the road, i.e., 
the edge of the road, and objects in his own lane are the most 
important information in the area surrounding the vehicle. 

Furthermore, it is advantageous to provide a deviation mirror 
that is semi-transparent in the came^ra device. One beam path, 
e.g., from the interior, may enter the camera device via 
reflection, and another beam path may enter via transmission 
through the semi-transparent mirror. As a result, there is no 
need for mechanical adjusting between the two beam paths . 
Furthermore, it is advantageous to design at least one 
deviation mirror to be concave or convex; as a result, the 
area that can be monitored by the camera can be limited or 
enlarged, depending on the use of the device. 
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Furthermore, it is advantageous to design the camera as a CCD 
camera or a CMOS camera. As a result, the camera device 
according to the present invention can be designed 
particularly inexpensively. Furthermore, it is advantageous to 
5 equip the camera device with at least two cameras, so that 
stereoscopic image capturing is possible, and so that 
conclusions can be drawn regarding the distances between the 
vehicle and objects and distances in the interior, 
respectively, by evaluating distance-dependent image shift. 

,0 

In addition, it is advantageous to arrange the camera device 
in an upper part of the windshield or to integrate the camera 
device into the roof of the vehicle. A position at least close 
to the vehicle roof allows an especially good overview of the 
:15 area surrounding the vehicle and of the vehicle's interior. 

Furthermore it is advantageous to design at least one 
deviation mirror so that it can be adjusted by an adjustment 
device so that at least the eyes and/or lips of the driver can 
20 be captured by the camera. This is particularly useful if 

drivers alternate and are of different heights and may arrange 
the seat in different positions. Furthermore, it enables the 
driver's movements while driving to be taken into account. By 
designing the deviation mirror so that the visible range 
25 captured can be readjusted, it is possible to ensure that the 
driver's eyes and/or lips are always within the capturing area 
of the camera device. This ensures that the means for 
monitoring whether the driver has fallen asleep and the means 
for checking speech input function properly, especially during 
30 driving. 

Drawings 

Exemplary embodiments of the present invention are shown in 
the drawings and are described in greater detail below. Figure 
35 1 shows an arrangement of the device according to the present 

invention in a motor vehicle; Figure 2 shows a flow chart of 
the method according to the present invention; Figures 2a and 
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2b show details of the method according to the present 
invention; Figure 2^shows an evaluation method according to 
the present invention; Figure 3 shows a flow chart for a 
further embodiment of the method according to the present 
invention; Figure :4_shows an embodiment of the device 
according to the present invention; Figure 5 shows another 
embodiment of the device according to the present invention; 
Figure 6 shows another embodiment of the device according to 
the present invention; Figure JT^shows a further embodiment of 
the device according to the present invention; and Figures ^8a 
and 8b_^how embodiments of a deviation mirror according to the 
present invention . 

Description of the Exemplary Embodiments 

In Figure 1, a camera device 10 according to the present 
invention is arranged in a motor vehicle on upper edge 11 of a 
windshield 12. The camera device has a first optical opening 
13, a first beam path 14 leading to a driver 15 of the 
vehicle. The mid-point beam of the beam path is shown. In 
addition, camera device 10 has a second optical opening 16, 
which is arranged on the opposite side of camera device 10 to 
first optical opening 13 and is therefore not visible in the 
view shown in the drawing. Second optical opening 16 is 
therefore only shown using a broken line. In addition, a 
second beam path 17, which leads from second optical opening 
16 of camera device 10 through windshield 12 into the area 
surrounding the vehicle in front to the vehicle, is shown. The 
driver's line of sight, which is shown as a third beam path 
18, extends in the same direction. Furthermore, cockpit 19 of 
the vehicle has a steering wheel 20 and a display unit 21. 
Herein, display unit 21 is preferably embodied as a 
combination instrument in which a plurality of displays are 
integrated into one electronic unit. In particular, a freely 
programmable combination instrument, in which various display 
instruments are shown on a screen, e.g., in the form of a 
liquid crystal display, is feasible., The figure also includes 
a processing unit which processes the image information 
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recorded by camera device 10, but this is not shown 
separately. The processing unit may either be arranged in the 
housing of camera device 10 shown, or in the vehicle roof on 
the other side of upper edge 11 of the windshield, or in 
cockpit 19 of the vehicle. In a preferred exemplary 
embodiment, the processing unit is arranged in a part of 
display unit 21 that is not visible to driver 15. As display 
unit 21 is used to output visual warning signals that are 
based on the evaluation by the processing unit of the image 
information recorded by camera device 10, e.g., if the driver 
is about to fall asleep or has exceeded the maximum speed 
limit, long data transmission paths can be avoided. 

Camera device 10 is arranged in the upper part of windshield 
12 close enough to the vehicle roof (not shown) so that the 
vehicle interior and the road in front of the vehicle can be 
monitored effectively. Preferably the camera device therefore 
is arranged in the middle of the vehicle with respect to the 
sides of the vehicle. It is also feasible for it to be 
arranged in the left upper part of windshield 12 in a left- 
hand-drive vehicle, as this ensures that not only the driver 
but also the entire road can be effectively captured by the 
camera device. In a right-hand-drive car, the camera is 
arranged in a right upper section of windshield 12. First and 
second optical openings 13, 16 may be designed in various 
ways. Any of the following ways are feasible: A filter, an 
opening, a lens, or a combination thereof in which the 
aforementioned components are arranged behind one another. 

Figure 2 shows a flow chart for the method according to the 
present invention. Starting from an initialization step 30, i 
a first process step 31 first image information 32 of the 
surrounding area is captured and evaluated by the processing 
unit, a first output 33 being output via visual and/or 
acoustic output media based on first image information 32. 
Thus the first output is based on the area surrounding the 
vehicle. In a subsequent second process step 34, image 
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information regarding the vehicle interior is determined from 
second image information 35, in which superimposed image 
information regarding the surrounding area and the vehicle 
interior is captured, based on first image information 32 
regarding the surrounding area obtained previously, by 
subtracting first image information 32 from second image 
information 35, so that based on the image information 
obtained a second output 36 is also output via visual and/or 
acoustic output media, the second output being in particular 
dependent on the image information regarding the vehicle 
interior. In a subsequent decision step 37, the process is 
aborted if the camera device is deactivated, i.e., in 
particular if the vehicle is turned off. This decision path is 
shown as Y in the drawing. In this case, the process ends when 
the camera device is switched off in a subsequent process step 
38. If the vehicle is not turned off, processing branches back 
to first process step 31. This decision path is shown as N in 
Figure 2 . 

In Figure 2a, first process step 31 is shown in detail. In a 
first sub-step 40, the camera device is switched on and first 
image information 32 is recorded. In a second sub-step 41, 
first image information 32 is sent to the processing unit for 
further processing. 

In Figure 2b, second process step 34 is subdivided into sub- 
steps. In a first sub-step 42, the radiation source that is 
not visible to the human eye is switched on by being supplied 
with electrical voltage. In a second sub-step 43, camera 
device 10 is switched on and a superimposed image of the 
interior and of the surrounding area is captured as second 
image information 35. In addition, a lighting adjustment must 
be carried out based on the lighting conditions, e.g., via an 
adjustable diaphragm opening or adjustment of the current 
applied to the light-sensitive sensors of the camera device. 
In a third sub-step 44, after the image has successfully been 
recorded, second image information 35 is stored and sent to 
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the processing unit for further processing. In a fourth sub- 
step 45, the radiation source that is not visible to the human 
eye is switched off. The image of the interior is then 
determined in a processing step (not shown in Figure 2b) in 
the processing unit. 

Figure 2c shows an evaluation process carried out by the 
processing unit which includes processing of the image 
information recorded by the camera device and of first output 
33 and second output 36, respectively. An example of an 
evaluation process is a falling-asleep warning generated by 
monitoring driver 15, monitoring of the vehicle's interior 
being necessary and second output 36 consequently being 
output. A method for detecting the surrounding area, e.g., for 
detecting road signs and/or road markings, can be embodied in 
a similar manner, first output 33 being output. 

In a first initialization step 50, the processing unit obtains 
an image of the driver's eye section from first and second 
image information 32 and 35. In a first decision step 52, the 
recorded image is compared with image information 51 regarding 
the driver's eye section that has been stored previously. 
Herein, image information 51 is an empty image if the vehicle 
has just been started up and as yet no image information has 
been stored. If it is determined that the driver's eyes are 
open, i.e., the driver is not asleep, or if image information 
51 is an empty image, processing branches along decision path 
N, and in process step 53 the recorded partial image is 
stored. Furthermore, the fact that the driver is awake at the 
time the image was recorded is stored in another memory. The 
evaluation process is ended in a completion step 54. The 
evaluation process is started again the next time first and 
second image information 32 and 35, respectively, are 
transmitted to the processing unit. A new start is carried out 
each time the evaluation process ends provided the vehicle or 
the camera device have not been switched off. 
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If the processing unit determines that the driver's eyes are 
closed, processing branches from first decision step 52 to a 
second decision step 55 along decision path Y. Here, a check 
is performed to determine whether the driver's eyes were 
already closed the last time an image was recorded. If not, 
processing branches to a sub-step 56, where data is stored 
indicating that the driver's eyes are closed at the point in 
time the image was recorded. In a completion step 57, the 
evaluation process is ended. If the driver's eyes were already 
closed the last time an image was recorded, processing 
branches along decision path Y from second decision step 55 to 
a first warning step 58. This warning is an audible warning 
and/or a visual warning, preferably via display unit 21. 
Because a warning is not issued until a second image has been 
recorded and thus after second decision step 55, it is 
generally possible to avoid a situation where a warning is 
issued because, by chance, the image was taken exactly at the 
moment the driver blinked, thus causing camera device 10 to 
detect that the driver's eyes are closed. 

After first warning step 58, a third decision step 59, in 
which image information 67 regarding a further image of the 
driver's face section is taken into account, is carried out. 
If the driver's eyes have reopened, processing branches along 
decision path Y to a processing step 60, image information 67 
that has been newly recorded being stored. Furthermore, data 
indicating that the driver's eyes are open is stored in a 
memory. The evaluation process is ended in a subsequent 
completion step 61. However, if the driver's eyes are still 
closed, processing branches from third decision step 59 along 
decision path N to a second warning step 62. In second warning 
step 62, a significantly louder audible warning is issued than 
that issued in first warning step 58. In a fourth decision 
step 63, image information 68 regarding the driver's facial 
section is captured again and status 69 of a switch is 
queried. If it is determined that the driver's eyes are now 
open or if the driver operates the switch, processing branches 
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along decision path Y. In a first sub-step 64, data indicating 
that the driver's eyes are open is stored and the evaluation 
process is ended in a completion step 65. If it is not 
determined that the driver's eyes are open or if it is not 
determined that the switch has been triggered, processing 
branches along decision path N to a third warning step 66. A 
loud audible warning is now issued again, and the vehicle is 
decelerated, the hazard warning lights system and the brake 
lights being activated so that driverless driving is avoided. 
As there are circumstances in which the camera device cannot 
obtain an image of the driver's eyes, e.g., if he is wearing 
sunglasses, the process shown in Figure 2c can be deactivated. 

Furthermore, it is possible to increase the number of times 
the image information regarding the driver's eye section is 
queried before an appropriate warning step is carried out, so 
as to avoid incorrect issuing of warnings. Herein, the number 
of queries is based on how frequently image information 
regarding the interior is captured. The process shown in 
Figure 2c may also be used to monitor the vehicle's position 
relative to a road marking if, instead of capturing image 
information regarding the driver's facial section, image 
information regarding the road marking is captured and the 
vehicle's position relative to the road marking is evaluated. 

Figure 3 shows a further method according to the present 
invention for monitoring the area surrounding the vehicle the 
vehicle interior. The same reference numbers represent the 
same process elements as those in Figure 2. Following an 
initialization step 30, in a first process step 80 first image 
information 81 regarding the vehicle's surrounding area is 
determined, sent to the processing unit, and first output 33 
is output based on first image information 81. In second 
process step 82, second image information 83 regarding the 
interior is captured by the camera device and sent to the 
processing unit. Second output 36 is output based on the image 
information captured. During first process step 80, an 
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electro-optical light valve in the direction of the vehicle's 
surrounding area is opened. In second process step 82, an 
electronic light valve in the direction of the vehicle's 
interior is opened. After second process step 82, a decision 
step 37 is carried out. If the camera device is switched off, 
processing branches along decision path Y and the camera 
device is switched off in a subsequent process step 38. 
Otherwise, processing branches back to first process step 80 
via decision path N. Herein, in a preferred exemplary 
embodiment, in first and second process steps 80, 82 the light 
valve in question is only opened for 90% of the duration of 
the process step in question. This ensures that the two sets 
of image information to be recorded do not overlap. 
Particularly at low temperatures, this keeps the image 
information to be recorded from overlapping, as low 
temperatures may cause the liquid crystal's switching behavior 
to become sluggish. The evaluation process described in Figure 
2c can be applied directly to the first output and/or second 
output 35 in Figure 3. 

Figure 4 shows an embodiment according to the present 
invention of a camera device 10 having a processing unit 110. 
Camera device 10 is arranged in a housing in which a camera 
100, which is designed as a CCD camera or a CMOS camera, is 
arranged with a first lens 101. Light from a first deviation 
mirror 102 enters first lens 101. First deviation mirror 102 
is semi-transparent, so that a first beam path 103 from the 
vehicle's surrounding area passes through an opening 109 in 
the housing of camera device 10, then passes through first 
deviation mirror 102 and then through first lens 101 to camera 
100. Furthermore, a second beam path 108 from a second 
deviation mirror 104 travels to first deviation mirror 102. 
Second beam path 108 is deviated by first deviation mirror 102 
and travels to camera 100. Second beam path 108 travels from 
the vehicle interior and enters camera device 10 through a 
second lens 107. Before it reaches second deviation mirror 
104, it passes through an infra-red filter 106. Camera 100 is 
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connected to processing unit 110 via a first data circuit 111. 
Processing unit 110 includes a control unit 112 and an 
evaluation unit 113^ which are connected to one another via a 
second data circuit 114. Evaluation unit 113 is connected via 

5 a third data circuit 117 to sensors 116 and via a fourth data 
circuit 118 at least to audible and/or visual display elements 
119. Furthermore, control unit 112 is connected via a fifth 
data circuit 120 to camera 100 and via a sixth data circuit 
122 to a radiation source 121, which emits radiation that is 

0 invisible to the human eye. Radiation source 121 is arranged 
in a housing which is preferably designed as a reflector 123. 

First beam path 103 and second beam path 108 are denoted by 
the optical axis of the beam in question. Here and in Figures 
.5 5-7, only the midpoint beam, which represents the entire beam 
path, is shown. In front of lens 101, the optical axis of the 
two beam paths coincides. However, for the purposes of 
clarity, in Figure 4 and the subsequent figures we have shown 
the two beam paths in parallel. 

0 

Processing unit 110 and camera device 10 may also be arranged 
in a single housing near the vehicle roof i.e., near the upper 
edge of windshield 12. However, processing unit 110 and camera 
device 10 may also be arranged in different places within the 
5 vehicle. In a preferred exemplary embodiment, processing unit 
110 is integrated into display unit 21. 

In first process step 31 in Figure 2, an image of the 
vehicle's surrounding area is captured by camera 100 with the 

0 help of first beam path 103. Herein, the image captured 

depends on how camera device 10 is arranged in the vehicle, 
and also on the size of opening 109 in the housing of camera 
device 10, and also on the setting of first lens 101. Herein, 
opening 109 preferably has a transparent cover, e.g., a 

5 transparent plastic disk. Furthermore, a third lens may be 

arranged there. When second process step 34 is carried out, in 
first sub-step 42 radiation source 121 is switched on by 
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control unit 112 via sixth data circuit 122 for the duration 
of the time period during which the image is captured, this 
being accomplished by applying a voltage to radiation source 
121. Figure 4 does not show the voltage source. The beam, 
5 which is bundled by reflector 123, is radiated into the 

vehicle's interior. The beam that is radiated is invisible to 
the human eye. Preferably the radiation source is designed as 
an infra-red beam diode or an infra-red beam diode array that 
includes a plurality of infra-red beam diodes. If the interior 
10 of the vehicle is illuminated by radiation source 121, the 
infra-red radiation that is reflected in the vehicle's 
interior passes through second lens 107 along second beam path 
108 into camera device 10 and reaches infra-red filter 106. 

r-n This filter only allows through infra-red radiation, so that 

visible light from the vehicle interior does not reach camera 

jj; 100. Thus in particular it is possible for the vehicle 

interior to be captured independently of visible light. 
Illumination of the interior is thus only dependent on the 
intensity of radiation source 121. Thereafter, the filtered 

r:;|i20 infra-red radiation passes to second deviation mirror 104, 

then to first deviation mirror 102, then to first lens 101 and 

in into camera 100. Second deviation mirror 104 has an adjustment 

&J device 30. In the figure only a mounting 130 of this 

adjustment device is shown. An electric motor, a control unit 
25 and a power supply are not shown. With the help of this 

adjustment device, second deviation mirror 104 can be rotated 
about an axis of rotation 131 within a certain angular range. 
As a result, the area of the interior which is imaged by 
second lens 107 and via the second deviation mirror into 
30 camera 100 can be modified. This is particularly useful if a 
driver changes the position of his seat while driving and 
camera device 10 must continue to capture his facial section. 

Sensors 116 may be designed as, for example, seat sensors, 
35 which supply information as to whether a seat is occupied. If 
a seat sensor reports that a seat is unoccupied, the camera 
can check whether this is true or whether there is movement, 
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for example, on the seat indicating that the seat is in fact 
occupied. In such cases, an airbag is not deactivated and/or 
seat heating is not deactivated. Furthermore, sensors also 
mean input elements via which, for example, a falling-asleep 
5 warning can be deactivated if the driver is wearing sunglasses 
which mean that his eyes cannot be seen camera 100. Output 
units mean audible and/or visual warning elements that may be 
embodied as a loudspeaker, a warning light or a liquid crystal 
display. Evaluation unit 113 and control unit 112 may also be 
10 integrated in a device. Furthermore, control unit 112 controls 
the position of second deviation mirror 104 via a connection 
line (not shown) , based on instructions transmitted from 
evaluation unit 113 via second data circuit 114, If an object 
being monitored by camera device 10 threatens to move beyond 
|L5 the visible range, the processing unit can in this way modify 
I the visible range via the control means of the second 

^ deviation mirror. First data circuit 111 and fifth data 

I circuit 120 constitute a connection between camera device 10 

' and processing unit 110. Herein, first data circuit 111 is 

j20 used to transmit image information from camera 100 to 

processing unit 110, in particular to evaluation unit 113. 
Processing unit 110, in particular control unit 112, controls 
camera 100 via fifth data circuit 120. First data circuit 111 
and fifth data circuit 120 may also be combined as a single 
25 data circuit. 

Figure 5 shows a further exemplary embodiment according to the 
present invention of the device for monitoring a vehicle's 
surrounding area and the vehicle interior. Here and in the 

30 subsequent figures, once again the same reference numbers 

denote the same components. In Figure 5, second beam path 108 
leaves the housing of camera 10 after passing through infra- 
red filter 106. In Figure 5, in order to distinguish the 
housing of camera device 10 from infra-red filter 106, we have 

35 shown the former using a broken line. The embodiment shown in 
Figure 5 allows the camera device to be arranged parallel to 
the sectional plane and perpendicular to the vehicle roof. In 

NY01 336616 



I flip I HI ' II 



a preferred exemplary embodiment, in which camera device 10 is 
arranged perpendicular to the vehicle roof, the area around 
the camera as far as opening 109 is completely housed in the 
vehicle roof, while the area around the second deviation 
mirror protrudes into the vehicle interior, i.e., the 
sectional plane in the drawing is perpendicular to the vehicle 
roof. Aside from adjustment of second deviation mirror 104, 
essentially the optical properties of first lens 101 are used 
to produce an image in camera 100. 

In Figure 6, a further embodiment according to the present 
invention of the device for monitoring a vehicle's surrounding 
area the vehicle interior is shown. In this exemplary 
embodiment, camera 100 is arranged on a different side of 
first deviation mirror 102 from that in Figures 4 and 5. In 
this case, the light that follows first beam path 103 is 
reflected by first deviation mirror 102 onto camera 100. By 
contrast, the light that follows second beam path 108 is 
deviated by second deviation mirror 104, so that the beam 
passes through first deviation mirror 102, which is embodied 
as a semi-transparent mirror, and ultimately reaches camera 
100. Furthermore, in this exemplary embodiment reflector 123 
is integrated into the housing of camera device 10, thus 
saving space. However, radiation source 121 may also be 
arranged in a favorable position in the vehicle some distance 
away from camera device 10. In addition, a plurality of 
radiation sources may be provided in the vehicle to ensure the 
vehicle's interior is optimally illuminated. 

Figure 7 shows a device for carrying out the method according 
to the present invention described in Figure 3. Instead of 
opening 109, an electro-optical light valve in the form of a 
first liquid crystal cell 151 is placed in first beam path 
103. First liquid crystal cell 151 can be controlled by 
control unit 112 via a control line 150, so that it is 
possible to switch first liquid crystal cell 151 back and 
forth between a transmissive and an absorptive state. The 
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structure of the liquid crystal cell is not shown in detail in 
the drawing, nor is the power supply shown. Herein, first 
liquid crystal cell 151 may be embodied so that a liquid 
crystal between two glass substrates is arranged between two 
5 transparent electrodes and influences the polarizing direction 
of light in different ways based on an electrical field that 
is applied. By arranging polarizing films on the glass 
substrates, it is possible to establish a desired level of 
absorption of light based on the voltage applied to the 

10 transparent electrodes or, respectively, a predefined maximum 
transmission of light based on the glass substrate, the 
polarizers and the liquid crystal. Furthermore, a second 
liquid crystal cell 153 is provided, can be controlled by 
control unit 112 via a control line 152 and is arranged in 

15 second beam path 108. In first process step 31, first liquid 
crystal cell 151 is switched over to a transparent state and 
second liquid crystal cell 153 is switched over to an 
absorptive state. In this case, only the light from the 
vehicle'' s surrounding area enters camera 100 along first beam 

20 path 103. In second process step 34, first liquid crystal cell 
151 is then switched over to its absorptive state and second 
liquid crystal cell 153 is switched over to its transmissive 
state. In this case, light passes along second beam path 108 
through a third lens 154 and via second deviation mirror 104 

25 and first deviation mirror 102 into camera 100. In order to 
avoid overlap, an intermediate step in which both liquid 
crystal cells 151 and 153 are switched over to their 
absorptive state may be inserted between the two process 
steps. This is recommended in particular at low temperatures, 

30 because in such cases switching over of the liquid crystal may 
be subject to a delay and maximum absorption and transmission, 
respectively, is not reached until the electrical field has 
been present for some time. By contrast with the exemplary 
embodiments shown in Figures 4 to 6, in the device shown in 

35 Figure 7 visible light also enters camera 100 along second 
beam path 108 . 
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Furthermore, in the case of all the aforementioned exemplary 
embodiments, two closely adjacent cameras whose first and 
second beam paths are offset slightly relative to one another 
may be provided instead of single camera 100. This allows 
images to be captured stereoscopically . With the help of 
suitable calculations performed by evaluation unit 113, 
conclusions regarding the distances to individual objects can 
be drawn from the captured stereoscopic images. This is 
advantageous in the case of detection of objects, e.g., road 
signs . 

Figures 8a and 8b show exemplary embodiments of second 
deviation mirror 104. In Figure 8a, a second deviation mirror 

1041 is concave, and in Figure 8b a second deviation mirror 

1042 is convex. It is feasible to use either deviation mirror 
1041 or deviation mirror 1042 as a second deviation mirror 
104. Because the mirror is embodied in this way, the area 
visible to the camera can be modified. The mirror shown in 
Figure 8b can be used to enlarge the beam area, whereas the 
mirror shown in Figure 8a can be used to limit the beam area; 
this is accomplished thanks to the differing curvature of the 
respective mirrors . 
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Claims 



1. A method for monitoring the interior of and area 
surrounding a vehicle, in particular a motor vehicle, 
characterized in that in a first method step, at least part of 
the area surrounding the vehicle, preferably in the direction 
of travel, and in a second method step, at least part of the 
vehicle's interior, preferably parts of the driver's body, are 
captured by a camera device (10), the first and second method 
steps are carried out alternately, and image information 
obtained in this way is sent to a processing unit (110) and 
processed there. 

2. The method according to Claim 1, 

characterized in that in the second process step the interior 
of the vehicle is illuminated by a radiation source (121) that 
is at least largely invisible to the human eye, preferably an 
infra-red radiation source. 

3. The method according to Claim 2, 

characterized in that in the second process step the part of 
the interior visible to the camera device (10) is superimposed 
on the part of the surrounding area visible to the camera 
device (10), and the image of the interior is determined by 
subtracting the image of the exterior obtained in the first 
process step . 

4. The method according to Claim 1, 

characterized in that in the first process step only the part 
of the surrounding area visible to the camera device (10) is 
captured, and in the second process step only the part of the 
interior visible to the camera device (10) is captured. 

5. The method according to Claim 4, 

characterized in that switching back and forth between 
capturing the interior and the surrounding area is 

NY01 336616 



IHIIPli 'I 



accomplished via at least one light valve, which is preferably 
an electro-optical light valve (151, 153) . 

6. The method according to one of the preceding claims, 
characterized in that an image that is captured constitutes in 
each case only a partial area of the maximum image that can be 
captured by the camera, in particular image rows, image 
columns or image pixels, and switching back and forth between 
capturing this partial area of the interior and of the 
surrounding area is carried out, and the partial areas 
captured are processed by the processing unit (110), and then 
the next partial area is captured. 

7. The method according to one of the preceding claims, 
characterized in that the driver's face, in particular his 
eyes, are captured. 

8. The method according to one of the preceding claims, 
characterized in that road markings and/or the position of the 
vehicle relative to the road markings are captured. 

9. The method according to one of Claims 7 through 8, 
characterized in that the driver's face and/or the position 
of the vehicle relative to the road markings are evaluated to 
determine whether the driver's eyes are open and/or to 
determine whether the vehicle is moving beyond a predefined 
area of the markings, and a visual and/or audible warning is 
issued based on the evaluation. 

10. The method according to one of the preceding claims, 
characterized in that road signs are captured. 

11. The method according to one of the preceding claims, 
characterized in that the number of people in the vehicle 
and/or the seat occupancy is determined. 

12- The method according to Claim 11, 

NY01 336616 

' iinii I 



characterized in that if a given seat is empty or occupied by 
a child seat deployment of the corresponding airbag and/or a 
seat heater is blocked. 



13. The method according to one of the preceding claims, 
characterized in that the lip movements of a predefinable 
person in the vehicle, preferably the driver, are captured in 
order to support a speech input system. 

14. A device for carrying out the method according to one of 
the preceding claims, 

characterized in that a vehicle, preferably a motor vehicle, 
has a camera device (10) and a processing unit (110, 112, 
113), at least part of the interior of and at least part of 
the area surrounding a vehicle can be captured by the camera 
device (10), that the camera device (10) is connected to the 
processing unit (110, 112), and that the images captured can 
be transmitted to the processing unit (110, 113) . 

15. The device according to Claim 14, 

characterized in that a first beam path (103) of the camera 
device points in the direction of the road in front of the 
vehicle and a second beam path (108) points in the direction 
of the interior, preferably the driver. 

16. The device according to one of Claims 14 through 15, 
characterized in that an illumination unit (121) that emits 
radiation that is at least largely invisible to the human eye 
in particular infra-red radiation, is provided and can be 
controlled by the processing unit (110, 112). 

17. The device according to one of Claims 14 through 16, 
characterized in that an infra-red filter (106) is arranged i 
the camera device (10), preferably in the second beam path 
(108) in the direction of the vehicle interior. 

18. The device according to one of Claims 14 through 15, 

22 
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characterized in that at least one light valve (151, 153), 
preferably a liquid crystal cell, is arranged in the camera 
device ( 10 ) . 

19. The device according to one of Claims 14 through 18, 
characterized in that at least one deviation mirror (102) that 
is preferably semi-transparent is arranged in the camera 
device (10) . 

20. The device according to one of Claims 14 through 19, 
characterized in that at least one deviation mirror (104) is 
designed to be concave (1041) or convex (1042) . 

21. The device according to one of Claims 14 through 20, 
characterized in that the camera device has a single camera 
(100), which is preferably designed as a CCD camera or a CMOS 
camera . 

22. The device according to one of Claims 14 through 20, 
characterized in that the camera device (10) has at least two 
cameras for capturing images stereoscopically . 

23. The device according to one of Claims 14 through 22, 
characterized in that the processing unit (110, 113) is 
connected to visual output units and/or acoustic output units 
(119, 21) which issue a warning to the driver, in particular 
if his eyes are closed or, respectively, the vehicle is about 
to move beyond a marked area of the road. 

24. The device according to one of Claims 14 through 23, 
characterized in that the camera device (10) is arranged in an 
upper part of the windshield (12) or is integrated into the 
roof of the vehicle. 

25. The device according to one of Claims 14 through 24, 
characterized in that at least one deviation mirror (104, 
1041, 1042) can be adjusted by an adjustment device (130) so 
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that in an image of the vehicle interior captured by the 
camera device (10) at least the eyes and/or lips of the driver 
can be seen. 
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Abstract 



A method and device for capturing the area surrounding and 
interior of a motor vehicle is described. The device for 

5 carrying out the method includes a camera device (10) having a 
beam path (103) that points in the direction of the area 
surrounding the vehicle, in particular the road, and having a 
beam path (108) that points in the direction of the vehicle 
interior. A processing unit (110) controls and evaluates the 

0 image information obtained. 
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Ij^ i . ^ [10191/1661] 

COMBINED DECLARATION AND 
POWER OF ATTORNEY FOR PATENT APPLICATION 

As a below named inventor, I hereby declare that: 

My residence, post office address and citizenship are as stated below adjacent 

to my name. 

I believe I am the original, first and sole inventor (if only one name is listed 
below) or an original, first and joint inventor (if plural names are listed below) of the subject 
matter which is claimed and for which a patent is sought on the invention entitled 
METHOD AND DEVICE FOR MONITORING THE INTERIOR AND 
SURROUNDING AREA OF A VEHICLE and the specification of which: 

[ ] is attached hereto; 

[] was filed as United States Application Serial No. on 

. 19 and was amended by the Preliminary Amendment 

filed on , 19 . 

[X] was filed as PCT International Application Number PCT/DEQO/01426 . 

on the 5!!L day of May, 2000. 
[X] an English translation of which is filed herewith. 
I hereby state that I have reviewed and understand the contents of the above- 
identified specification, including the claims, as amended by any amendment referred to 
above. 

I acknowledge the duty to disclose information which is material to the 
examination of this application in accordance with Title 37, Code of Federal Regulations, 
§L56(a). I hereby claim foreign priority benefits under Title 35, United States Code § 1 19 of 
any foreign application(s) for patent or inventor's certificate or of any PCT intemafional 
applications(s) designating at least one country other than the United States of America listed 
below and have also identified below any foreign application(s) for patent or inventor's 
certificate or any PCT international application(s) designating at least one country other than 
the United States of America filed by me on the same subject matter having a filing date 
before that of the application(s) of which priority is claimed: 
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PRIOR FOREIGN/PCT APPLICATION(S) 

AND ANY PRIORITY CLAIMS UNDER 35 U.S.C. § 119 

Country : Federal Republic of Germany 

Application No. : 199 21 488.3 

Date of Filing: May 8, 1999 

Priority Claimed 

Under 35 U.S.C. §119: [X] Yes [ ] No 

I hereby claim the benefit under Title 35, United States Code § 120 of any United States 
Application or PCT International Application designating the United States of America that 
is/are listed below and, insofar as the subject matter of each of the claims of this application 
is not disclosed in that/those prior application(s) in the manner provided by the first paragraph 
of Title 35, United States Code § 1 12, 1 acknowledge the duty to disclose material 
information as defined in Title 37, Code of Federal Regulations § 1.56(a) which occurred 
between the filing date of the prior application(s) and the national or PCT international filing 
date of this application: 

PRIOR U.S. APPLICATIONS OR 
PCT INTERNATIONAL APPLICATIONS 
DESIGNATING THE U.S. FOR BENEFIT UNDER 35 U.S.C. § 120 

U.S. APPLICATIONS 

Number : 

Filing Date : 

PCT APPLICATIONS 
DESIGNATING THE U.S. 

PCT Number : 

PCT Filing Date : 

I hereby appoint the following attomey(s) and/or agents to prosecute the 
above-identified application and transact all business in the Patent and Trademark Office 
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connected therewith. 



(List name(s) and registration number(s)): 




Richard L. Mayer, 
Gerard A. Messina, 



Reg. No. 35^i- 

Reg. No. 

Reg. No. 



Reg: N o. 22,49 0 



All correspondence should be sent to 



Richard L. Mayer, Esq. 



Ke nyon & KcnyUTr~ - 

"One Broadway 

New York, New York 1QQ04 



Telephone No.: (212) 425-7200 
Facsimile No.: (212) 425-5288 

I hereby declare that all statements made herein of my own knowledge are true 



and that all statements made on information and belief are believed to be true and further that 
these statements were made with the knowledge that willful false statements and the like so 
made are punishable by fine or imprisonment or both under Section 1001 of Title 18 of the 
United States Code and that such willful false statements may jeopardize the validity of the 
application or any patent issuing thereon. 
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Full name of invent or Winfried KOENIG 

Inventor's signature 1/ yi^^cj USi^j Date j 

Citizenship Federal Republic of Germany 

Residence Murgstr. 8 

76327 Pfinztal 

Federal Republic of Germany 
Post Office Address Same as above 
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Full name of inventor Bernd HURTGE N 

Inventor's signature >^/^4^'^^>^ Date X /^^iPA 

Citizenship Federal Republic of Germany 

Residence Dietrich-Bonhoeffer-Str. 4 ^ 
31079,Sibbesse ' 
Federal Republic of Germany 

Post Office Address Same as above 




uJ 

m 
o 

m 

m 

Q 

Ml. 



Page 5 of 6 



Full name of inventor Werner POCHMULLER 

Inventor's signature _ ^I'fAiA^i^^mm Date ^lljOj 

Citizenship Federal Republic of Germany A^U^i^CiS^ 

Residence Gutcnbcrgstr. 19 OS'i<^rbriink // 
31139Hildesheim 
Federal Republic of Germany 

Post Office Address Same as above 
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